<!DOCTYPE html>
<html xmlns:MadCap="http://www.madcapsoftware.com/Schemas/MadCap.xsd" class="_Skins_HTML5___Top_Navigation" lang="en-us" xml:lang="en-us" data-mc-search-type="Stem" data-mc-help-system-file-name="Default.xml" data-mc-path-to-help-system="../../" data-mc-has-content-body="True" data-mc-target-type="WebHelp2" data-mc-runtime-file-type="Topic;Default" data-mc-preload-images="false" data-mc-in-preview-mode="false" data-mc-toc-path="">
    <head>
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <meta charset="utf-8" />
        <meta http-equiv="X-UA-Compatible" content="IE=edge" />
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <link href="../../Skins/Default/Stylesheets/Slideshow.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/TextEffects.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Topic.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Components/Styles.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Components/Tablet.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Components/Mobile.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/foundation.6.2.3.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/Styles.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/Tablet.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/Mobile.css" rel="stylesheet" type="text/css" data-mc-generated="True" /><title>Working with Datasets</title>
        <link href="../Resources/Stylesheets/Styles.css" rel="stylesheet" type="text/css" />
        <script src="../../Resources/Scripts/custom.modernizr.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/jquery.min.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/require.min.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/require.config.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/foundation.6.2.3_custom.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/plugins.min.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapGlobal.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapDom.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapUtilities.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapXhr.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapTextEffects.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapSlideshow.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapFeedback.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapDefault.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapHelpSystem.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.Breadcrumbs.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.MiniToc.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.SideMenu.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapIndex.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapGlossary.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapParser.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapSearch.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapTopic.js" type="text/javascript">
        </script>
    </head>
    <body>
        <div class="foundation-wrap off-canvas-wrapper">
            <div class="off-canvas-wrapper-inner" data-off-canvas-wrapper="">
                <aside class="off-canvas position-left" id="offCanvas" data-off-canvas="" data-position="left" data-mc-ignore="true">
                    <ul class="off-canvas-drilldown vertical menu off-canvas-list" data-drilldown="" data-mc-back-link="Back" data-mc-css-tree-node-expanded="is-drilldown-submenu-parent" data-mc-css-tree-node-collapsed="is-drilldown-submenu-parent" data-mc-css-sub-menu="vertical menu slide-in-left is-drilldown-submenu" data-mc-include-indicator="False" data-mc-include-icon="False" data-mc-include-parent-link="True" data-mc-include-back="True" data-mc-defer-expand-event="True" data-mc-expand-event="click.zf.drilldown" data-mc-toc="True">
                    </ul>
                </aside>
                <div class="off-canvas-content inner-wrap" data-off-canvas-content="">
                    <div data-sticky-container="" class="title-bar-container">
                        <nav class="title-bar tab-bar sticky" data-sticky="" data-options="marginTop:0" style="width:100%" data-sticky-on="only screen and (max-width: 1000px)" data-mc-ignore="true">
                            <div class="middle title-bar-section outer-row clearfix">
                                <div class="menu-icon-container relative clearfix">
                                    <button class="menu-icon" data-toggle="offCanvas"><span></span>
                                    </button>
                                </div>
                            </div>
                            <div class="title-bar-layout outer-row">
                                <div class="logo-wrapper"><a class="logo" href="index.htm" alt="Logo"></a>
                                </div>
                                <div class="navigation-wrapper nocontent">
                                    <ul class="navigation clearfix" data-mc-css-tree-node-has-children="has-children" data-mc-css-sub-menu="sub-menu" data-mc-expand-event="mouseenter" data-mc-top-nav-menu="True" data-mc-max-depth="3" data-mc-include-icon="False" data-mc-include-indicator="False" data-mc-include-children="True" data-mc-include-siblings="True" data-mc-include-parent="True" data-mc-toc="True">
                                        <li class="placeholder" style="visibility:hidden"><a>placeholder</a>
                                        </li>
                                    </ul>
                                </div>
                                <div class="nav-search-wrapper">
                                    <div class="nav-search row">
                                        <form class="search" action="#">
                                            <div class="search-bar search-bar-container needs-pie">
                                                <input class="search-field needs-pie" type="search" placeholder="Search" />
                                                <div class="search-filter-wrapper">
                                                    <div class="search-filter">
                                                        <div class="search-filter-content">
                                                            <ul>
                                                                <li>All Files</li>
                                                            </ul>
                                                        </div>
                                                    </div>
                                                </div>
                                                <div class="search-submit-wrapper" dir="ltr">
                                                    <div class="search-submit" title="Search">
                                                    </div>
                                                </div>
                                            </div>
                                        </form>
                                    </div>
                                </div>
                            </div>
                        </nav>
                    </div>
                    <section class="main-section">
                        <div class="row outer-row sidenav-layout">
                            <div class="sidenav-wrapper">
                                <div class="sidenav-container">
                                    <ul class="off-canvas-accordion vertical menu sidenav" data-accordion-menu="" data-mc-css-tree-node-expanded="is-accordion-submenu-parent" data-mc-css-tree-node-collapsed="is-accordion-submenu-parent" data-mc-css-sub-menu="vertical menu accordion-menu is-accordion-submenu nested" data-mc-include-indicator="False" data-mc-include-icon="False" data-mc-include-parent-link="False" data-mc-include-back="False" data-mc-defer-expand-event="True" data-mc-expand-event="click.zf.accordionMenu" data-mc-toc="True" data-mc-side-nav-menu="True">
                                    </ul>
                                </div>
                            </div>
                            <div class="body-container" data-mc-content-body="True">
                                <h1>Working with Datasets</h1>
                                <p>The section covers the following topics:</p>
                                <ul>
                                    <li value="1"><a href="#Uploadin" class="MCXref xref">Uploading Datasets</a>
                                    </li>
                                    <li value="2"><a href="#nctl" class="MCXref xref">nctl mount Command</a>
                                    </li>
                                    <li value="3"><a href="#Mount" class="MCXref xref">Mount and Access Folders</a>
                                    </li>
                                    <li value="4"><a href="#Uploadin2" class="MCXref xref">Uploading and Using Shared Dataset Example</a>
                                    </li>
                                </ul>
                                <h2><a name="Uploadin"></a>Uploading Datasets</h2>
                                <p>Nauta offers two ways for users to upload and use datasets for experiments:</p>
                                <h4>Uploading During Experiment Submission</h4>
                                <p>Uploading additional datasets or files is an option available for the ‘submit’ command, using the following option:</p>
                                <p style="text-indent: 0.5in;">-<span style="font-family: 'Courier New';">sfl, --script_folder_location</span></p>
                                <p>Where <span style="font-family: 'Courier New';">script_folder_location</span> is the name of a folder with additional files used by a script, e.g., other .py files, datasets, etc. If the option is not included, the files will not be included in the experiment.</p>
                                <p><b>Syntax</b>:</p>
                                <p><span style="font-family: 'Courier New';">nctl experiment submit --script_folder_location DATASET-PATH SCRIPT_LOCATION</span>
                                </p>
                                <p>This option is recommended for small datasets or datasets that are NOT reused often. In using this option, the dataset will be uploaded each time the submit command is executed which may add to the overall submission time.</p>
                                <p><b>Note</b>: <span style="font-family: 'Courier New';">script_folder_location</span> is also used for script files, not only for datasets.</p>
                                <h4>Upload to Nauta storage and Reference Location</h4>
                                <p>Depending on the Nauta configuration, the application uses NFS to connect to a storage location where each user has folders that have been setup to store experiment input and output data. This option allows the user to upload files and datasets for private use and for sharing. Once uploaded, the files are referenced by the path. All data in the folders are retained until the user manually removes it from the NFS storage. Refer to the instructions in this chapter for information how to access and use Nauta storage.</p>
                                <p>This option is recommended for large datasets, data that will be reused often, and data that will be shared among users.</p>
                                <h2><a name="nctl"></a>nctl mount Command</h2>
                                <p>The ‘mount’ command displays another command that can be used to mount Nauta folders to a user’s local machine. When a user executes the command, information similar the following is displayed (this example is for macOS).  Use the following command to mount those folders (all of the following is displayed, although this is an example only).</p>
                                <p style="text-align: center;">
                                    <img src="../images/nctl_mount_command_558x451.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 558;height: 451;" />
                                </p>
                                <p><b>Note 1</b>: The <span style="font-family: 'Courier New';">nctl mount</span> command also returns a command to <i>unmount</i> a folder.</p>
                                <p><b>Note 2</b>: Nauta uses the mount command that is native to each operating system, so the command printed out may not appear as in the example.</p>
                                <p><b>Note 3</b>: All variables are shown in upper-case letters.</p>
                                <h2><a name="Mount"></a>Mount and Access Folders</h2>
                                <p>The following table displays the access permissions for each mounting folder.</p>
                                <table style="width: 100%;border-collapse: collapse;border-top-left-radius: 1px;border-top-right-radius: 1px;border-bottom-right-radius: 1px;border-bottom-left-radius: 1px;border-left-style: solid;border-left-width: 2px;border-right-style: solid;border-right-width: 2px;border-top-style: solid;border-top-width: 2px;border-bottom-style: solid;border-bottom-width: 2px;">
                                    <caption>&#160;</caption>
                                    <col style="width: 227px;" />
                                    <col style="width: 254px;" />
                                    <col style="width: 256px;" />
                                    <col style="width: 257px;" />
                                    <thead>
                                        <tr>
                                            <th style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>NAUTA Folder</p>
                                            </th>
                                            <th style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>Reference Path</p>
                                            </th>
                                            <th style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>User Access</p>
                                            </th>
                                            <th>
                                                <p>Shared Access</p>
                                            </th>
                                        </tr>
                                    </thead>
                                    <tbody>
                                        <tr>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>input</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>/mnt/input/home</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>read/write</p>
                                            </td>
                                            <td>&#160;</td>
                                        </tr>
                                        <tr>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>output</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>/mnt/output/home</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>read/write</p>
                                            </td>
                                            <td>&#160;</td>
                                        </tr>
                                        <tr>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>input-shared</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>/mnt/input/root/public</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>read/write</p>
                                            </td>
                                            <td>read/write</td>
                                        </tr>
                                        <tr>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>output-shared</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>/mnt/output/root/public</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>read/write</p>
                                            </td>
                                            <td>read/write</td>
                                        </tr>
                                        <tr>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>input-output-ro</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>&#160;</p>
                                            </td>
                                            <td style="border-left-style: solid;border-left-width: 1px;border-right-style: solid;border-right-width: 1px;border-top-style: solid;border-top-width: 1px;border-bottom-style: solid;border-bottom-width: 1px;">
                                                <p>read</p>
                                            </td>
                                            <td>read</td>
                                        </tr>
                                    </tbody>
                                </table>
                                <h2><a name="Uploadin2"></a>Uploading and Using Shared Dataset Example</h2>
                                <p>The default configuration is to mount my-input and my-output folders to Nauta storage. Perform these steps below to mount an input-shared folder to Nauta storage so that you and other users can use shared input data when performing training.</p>
                                <ol>
                                    <li value="1">Linux/macOS only: Create a folder for mounting named my-input.: <br /><span style="font-family: 'Courier New';">$ mkdir my-input</span><br /></li>
                                    <li value="2">Use the mount command to display the command that should be used to mount your local folder/machine to your Nauta input folder.<br /><span style="font-family: 'Courier New';">$ nctl mount</span><br /></li>
                                    <li value="3">
                                        <p>Use the mounting command that was provided to mount your local machine to Nauta storage. Enter the mount command that is provided by <span style="font-family: 'Courier New';">nctl mount</span> using the input-shared as the NAUTA_FOLDER and my-shared-input folder or ‘Y:’ as the MOUNTPOINT. Here are examples of mounting the local folder to the Nauta output folder for each OS:</p>
                                        <ul>
                                            <li value="1">MacOS:&#160;m<span style="font-family: 'Courier New';">ount_mbfs //'USERNAME:PASSWORD'@CLUSTER-URL/input- my-input</span></li>
                                            <li value="2">Ubuntu:&#160;<span style="font-family: 'Courier New';">sudo mount.cifs  -o username=USERNAME,password=PASSWORD,rw,uid=1000 //CLUSTER-URL/input my-input</span></li>
                                            <li value="3">Windows: Use Y: drive as mount point<br /><span style="font-family: 'Courier New';">net case Y: \\CLUSTER-URL\input /user:USERNAME PASSWORD</span><br /></li>
                                        </ul>
                                    </li>
                                    <li value="4">
                                        <p>Navigate to the mounted location:</p>
                                        <ul>
                                            <li value="1">MacOS/Ubuntu only: Navigate to my-shared-input folder.</li>
                                            <li value="2"> Windows: Open Explorer Window and navigate to Y: drive</li>
                                        </ul>
                                    </li>
                                    <li value="5">Copy input data or files to this folder for use when submitting experiments.  After copying, the files will be located in Nauta storage and can be used by any user for their experiments. </li>
                                    <li value="6">Using the mnist example in <a href="getting_started.htm" class="MCXref xref">Getting Started</a>, you can download the mnist dataset from this link:<br />Mnist Dataset: <a href="experiment.htm">http://yann.lecun.com/exdb/mnist</a></li>
                                    <li value="7">Create a <span style="font-family: 'Courier New';">mnist</span> folder in the Nauta input folder. </li>
                                    <li value="8">Copy the downloaded files to the folder.</li>
                                    <li value="9">Submit an experiment referencing the new shared dataset. From the examples folder:<br />:<span style="font-family: 'Courier New';">$ nctl experiment submit --name mnist-shared-input  mnist_with_summaries.py -- --data_dir==/mnt/input/home/mnist</span></li>
                                    <li value="10">If you want to to copy your data to shared folder use input-shared instead of input in step 2. Doing so lets any Nauta user can use the same path to reference the mnist dataset on your shared Nauta&#160;storage<p>&#160;</p></li>
                                </ol>
                                <p>&#160;</p>
                            </div>
                        </div>
                    </section><a data-close="true"></a>
                </div>
            </div>
            <script>/* <![CDATA[ */$(document).foundation();/* ]]> */</script>
        </div>
    </body>
</html>